
SECURITY APPARATUS 
AND METHOD 

Background of the 
Invention 

[0001] Field of the Invention: 
[0002] The present invention 
relates to a security apparatus and a 
method for supplying a service to a 
user by confirming a use situation of 
the user and infringement situation of 
non-user in order to keep safety and 
reliability for the service. 
[0003] Description of the Related 
Art: 

[0004] A method to specify the 
user in service-use area by electrical 
means and to execute security for each 
kind of service is very important in 
present electrical society. Various 
kinds of security means, for example 
memorized number, magnetic card, IC 
card, wireless card, sound comparison, 
fingerprint comparison, retina 
comparison and face comparison, have 
been used. However, the memorized 
number, the magnetic card, the IC card 
and the wireless card specifies the user 
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by information only the user knows or 
with a memory device key for only the 
user. Therefore, they are in danger of 
being used unjustly. 
[0005] On the other hand, sound 
comparison, fingerprint comparison, 
retina comparison, and face 
comparison are effective personal 
confirmation means to detect [defect] 
an other's unjust use because the 
user's physical features are used as a 
key. However, the personal 
confirmation is only used at the 
beginning of service. After the service 
begins, the service can not be 
interrupted. Therefore [Therfore], 
even if an other person changes for the 
user after the service is begun, the 
service can not be interrupted. In 
short, this kind of service includes 
danger that the other person utilizes 
the service unjustly [unjusty]. This 
kind of problem is included in the 
memorized number, the magnetic card, 
the IC card and the wireless card. 
[0006] As mentioned [mentiored] 
above, the known security apparatuses 
include the following problems. 
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[0007] (1) After security is 
cancelled by confirming the user, 
security function is not executed while 
supplying the service. Therefore, even 
if an other person changes for the user, 
the service is still supplied 
continuously. 

[0008] (2) While supplying the 
service to the user, it often happens 
that an other person watches 
information not to be disclosed on 
display from behind the user. This 
problem occurs because security is not 
provided in the physical space 
surrounding the user and the apparatus 
even if security is executed inside the 
apparatus electrically. Especially, this 
problem happens when the service is 
supplied to the user in public place in 
which other persons are not excluded. 

Invention Summary 

[0009] It is an object of the present 
invention to provide a security 
apparatus and method to supply a 
service to the user while safety and 
reliability for the service are kept. 



[0010] According to the present 
invention, there is provided a security 
apparatus for a device supplying a 
service to a user, comprising: person 
discrimination means for recognizing a 
user requesting the service, use 
situation decision means for deciding 
whether the user is under a situation to 
use the service, infringement situation 
decision means for detecting whether a 
non-user intrudes into a use area, the 
service to decide whether of service is 
infringed, service control means for 
supplying the service to the user in 
case said person discrimination means 
recognizes the user, and for controlling 
a supply of the service if said use 
situation decision means decides the 
user is not under the situation to use 
the service or if said infringement 
situation decision means decides that 
the security of the service is infringed. 
[0011] Further in accordance with 
the present invention, there is provided 
a security method associated with 
supplying a service to a user, 
comprising the step of: recognizing a 
user desiring the service: supplying the 
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service to the user if the user is 
recognized, deciding whether the user 
is under a situation to use the service, 
deciding whether a non-user intrudes 
into a use area of the service, and 
controlling the supply of the service if 
the user is not under the situation to 
use the service or if the non-user is 
intruding into the use area of the 
service. 

[0012] Further in accordance with 
the present invention, there is also 
provided a computer readable memory 
containing computer-readable 
instructions to supply a service to a 
user, comprising: instruction means for 
causing a computer to recognize a user 
desiring the service, instruction means 
for causing a computer to supply the 
service to the user if the user is 
recognized, instruction means for 
causing a computer to decide whether 
the user is under a situation to use the 
service, instruction means for causing 
a computer to decide whether a non- 
user intrudes into a use area of the 
service, and instruction means for 
causing a computer to control a supply 



of the service if the user is decided not 
under the situation to use the service or 
the non-user is decided to intrude into 
the use area of the service. 

Brief Description of the 
Drawings 

[0013] FIG. 1 is a block diagram 
of the security apparatus according to 
the present invention. 
[0014] FIG. 2 is a block diagram 
of the situation detection section of the 
security apparatus according to the 
present invention. 

[0015] FIG. 3 is a flow chart of 
processing of a service supply 
permission section according to a first 
embodiment of the present invention. 
[0016] FIG. 4 is a flow chart of 
processing of the service supply 
permission section according to a 
second embodiment of the present 
invention. 

[0017] FIG. 5 is a flow chart of 
processing of the service supply 
permission section according to a third 
embodiment of the present invention. 
[0018] FIG. 6 is a flow chart of 
processing of the service supply 
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permission section according to a 
fourth embodiment of the present 
invention. 

[0019] FIG. 7 is a block diagram 
of the security [sequrity] apparatus 
according to a fifth embodiment of the 
present invention. 

[0020] FIG. 8 is a flow chart of 
processing of the service supply 
permission section according to the 
fifth embodiment of the present 
invention. 

[0021] FIG. 9 is a flow chart of 
processing of the service supply 
permission section according to a sixth 
embodiment of the pres sent invention. 
[0022] FIG. 10 is a flow chart of 
processing of the service supply 
permission section according to a 
seventh embodiment of the present 
invention. 

[0023] FIG. 1 1 is a block diagram 
of the security [sequrity] apparatus 
according to a eighth embodiment of 
the present invention. 
[0024] FIG. 12 is a flow chart of 
processing of the service supply 
permission section according to the 



eighth embodiment of the present 
invention. 

[0025] FIG. 13 is a schematic 
diagram of detection of visual lines of 
the user in the image according to the 
eighth embodiment of the present 
invention. 

[0026] FIG. 14 is a flow chart of 
processing of the service supply 
permission section according to a ninth 
embodiment of the present invention. 
[0027] FIG. 15 is a schematic 
diagram of detection of visual lines of 
the non-user in the image according to 
the ninth embodiment of the present 
invention. 

[0028] FIG. 16 is a flow chart of 
processing of the service supply 
permission section according to a tenth 
embodiment of the present invention. 
[0029] FIG. 17 is a schematic 
diagram of supply method for the 
security apparatus by software 
according to the present invention. 

Detailed Description of the 
Preferred Embodiments 

[0030] The embodiments of the 
present invention are described below 
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with reference to the drawings. FIG. 1 
is a block diagram of the security 
apparatus according to the present 
invention. The security apparatus of 
the present invention is comprised of a 
situation detection section 1, a service 
supply permission section 2, and a 
service supply section 3. The situation 
detection section 1 discriminates the 
user in service-use areas, detects 
whether the user is under a situation to 
use the service (use situation) and 
detects whether a non-user infringes 
the security of the user (infringement 
situation). The detection result of the 
use situation and the infringement 
situation is outputted to the service 
supply permission section 2 as person 
information. In this place, the 
situation that the user can utilize the 
service represents that the user is 
confirmed to be permitted the supply 
of the service and the user 
continuously utilizes the service. A 
situation that the user continuously 
utilizes the service represents a status 
that the user actually interacts with the 
service application (the user inputs 



operation to system or the user is 
supplied output from the system). A 
status of non-interaction within 
predetermined time indicates thinking, 
chattering, looking away. Another 
status indicates when the user moves 
away from service use-space within a 
predetermined time. 
[0031] The situation that a non- 
user infringes the security of the user 
represents that a person other than the 
confirmed user intentionally or 
accidentally uses the service while the 
service is supplied to the confirmed 
user. The other person can then 
interact with the operation or 
information to be disclosed only to the 
confirmed user. When the user 
interacts with the service application of 
the system through a display, the other 
person looks on the display behind the 
user. Otherwise, the other person 
intends to interact with the service 
application in the user's absence 
within the predetermined time. 
[0032] In order to detect each 
situation, the situation detection 
section 1 monitors service-use area 
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using image or sound. Especially, the 
use situation of the user and the 
infringement situation of the security 
are detected by the image (person 
detection/direction of face 
detection/person confirmation) and the 
sound (person detection/person 
specification/ chattering detection). 
Monitoring may be wireless. 
[0033] The service supply 
permission section 2 controls 
start/interruption/interruption and 
cancellation/end of the service for the 
service supply section 3 according to 
the detection result of the situation 
detection section 1. In this case, the 
service supply permission section 2 
starts the supply of the service when 
the user is specified in the service-use 
area. The service supply permission 
section 2 controls the supply of the 
service in at least one of the cases 
where the user does not utilize the 
service or where the security of the 
user is infringed. Especially when the 
user does not utilize the service during 
the supply of the service, the supply of 
the service finishes If the security of 




the user is infringed by another person 
the supply of the service interrupts 
[interupts] until the infringement is 
relieved. The service supply section 3 
supplies various kinds of service under 
the control of the service supply 
permission section 2. For example an 
uninhabited ATM (automatic 
[antomatic] teller's machine), a 
portable terminal, or a computer used 
by several persons may comprise the 
service supply section 3. 
[0034] FIG. 2 is a block diagram 
of the situation detection section 1 . In 
this case, an image is used as a way to 
discriminate the user, detecting the use 
situation, or detect an infringement 
situation. The situation detection 
section 1 is comprised of an image 
input section 11, a person area 
detection section 12, a person 
comparison section 13, a use situation 
decision section 14 and an 
infringement situation decision section 
15. The image input [imput] section 
11 inputs an image through a TV 
camera. The TV camera is set in the 
service use area for continuously 



Library: Los Angeles; Document #: 40 1 65 v I 
43701.00023 



-6- 



observing the user to be supplied the 
service. The person area detection 
section 12 analyzes images input by 
the image input section 11 and 
monitors the secure area from the input 
image. For example, a difference 
between the input image and 
background image (uninhabited area) 
indicates a person present in the secure 
area. The person area detection 
section 12 outputs person detection 
information Dl when the person 
present is detected and outputs non- 
detection information Dl when the 
person present is not detected. 
[0035] The person comparison 
section 13 compares the person present 
area image with the content of a person 
comparison dictionary to specify 
which person is present in the secure 
area image. The person comparison 
dictionary previously stores at least 
comparison information of persons 
permitted to use the service. The 
person comparison section 13 outputs 
person discrimination information D2 
representing whether the person 
present coincides with a person 



permitted to use the service. If the 
person present coincides with a person 
permitted to use the service in the 
person comparison dictionary, the 
person present is registered as the user 
who can be supplied the service. 
[0036] The use situation decision 
section 14 detects whether the user is 
under a situation capable of using the 
service. After the user of the person 
area is registered and if the registered 
user is now included in the person 
discrimination information D2, the 
user is decided to be capable of using 
the service. The use situation decision 
[decision] section 14 outputs use 
situation information D3 representing 
whether the user is capable of using 
the service. 

[0037] The infringement situation 
decision section 15 detects whether a 
non-user infringes the security of the 
service. If a person other than the user 
utilizing the service now is included in 
the person discrimination information 
D2, the infringement situation decision 
section 15 decides that the security of 
the service is infringed, and outputs 
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infringement situation information D4 
indicates a breach of security. The 
person detection/non-detection 
information Dl, the person 
discrimination information D2, the use 
situation information D3, the 
infringement situation information D4 
are called person information. The 
person information is read out by the 
service supply permission section 2 
(step SI in FIG. 3 explained afterward) 
and referred at steps S3, S4, S8, SI 1 in 
FIG. 3. 

[0038] FIG. 3 is a flow chart of 
processing of the service supply 
permission section 2. As mentioned 
above, the situation detection section 1 
discriminates the user in the service- 
use area and monitors the use situation 
and the infringement situation using 
the image inputted from the TV 
camera. The person information as 
detection result of the situation 
detection section 1 is outputted to the 
service supply permission section 2. 
By using the person information, the 
service supply permission section 2 
executes a security controlling process 



to the service supply section 3. First, 
the service supply permission section 2 
obtains the person information from 
the situation detection section 1 (step 
SI). When the supply of the service 
begins ("No" at step S2), the service 
supply permission section 2 monitors 
the security area in the service use area 
according to the person detection/non- 
detection information Dl (step S3). If 
a person area is not detected ("No" at 
step S3), processing of the service 
supply permission section 2 is returned 
to step SI. On the other hand, if a 
person is detected ("Yes" at step S3), 
the service supply premission section 2 
decides whether the person is 
permitted as a user (step S4). As a 
result, in case the person is not 
permitted as a user ("No" at step S4), 
the service supply permission section 2 
rejects the supply of the service (step 
S5) and returns processing to step SI. 
In case the person is permitted as a 
user ("Yes" at step S4), the service 
supply permission section 2 registers 
the person as the user (step S6). After 
that, the use situation of the person is 
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continuously monitored. The service 
supply permission section 2 begins the 
supply of the service (step S7) and 
returns the processing to SI . 
[0039] The above process steps 
(step S1-S7) are the same as normal 
confirmation of security (from 
confirming the user to beginning 
[beginning] the supply of the service). 
In this place, after the supply of the 
service is begun, the registered user 
exists in the service-use area and the 
registered person must be detected at 
step SI After the supply of the service 
is begun, the processing is returned to 
step S8 through steps SI and S2. 
[0040] The service supply 
permission section 2 decides whether 
the registered person is using the 
service according to the use situation 
information D3 (step S8). As a result, 
if the situation of using the service is 
detected, for example, a user's absence 
continues above a predetermined time 
("No" at step S8), the service supply 
permission section 2 terminates the 
supply of the service immediately (step 
S9). Next, the registration of the user 



is cancelled (step S10) and the 
processing returns to step SI in order 
to register a new user. 
[0041] On the other hand, the 
service supply permission section 2 
decides whether a non-user infringes 
the security of the user according to 
the infringement situation information 
D4 (step Sll). As a result, if the 
situation that a non-user infringes the 
security is detected, for example, an 
other person watches the display 
behind the user during the supply of 
the service ("Yes" at step Sll), the 
service supply permission section 2 
temporarily interrupts the service to 
maintain security until the security is 
not infringed (step SI 2). If a situation 
that the security is not infringed is 
detected during the interruption of the 
service ("Yes" at step SI 3), the service 
supply permission, section 2 cancels 
the interruption of the service and 
begins the supply of the service (step 
S14). 

[0042] In this way. in case the user 
permitted the service is specified in the 
service use area, the supply of the 
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service is begun. During the supply of 
the service, if the user is not using the 
service or if the security is breached, 
the supply of the service finishes or 
interrupts. Accordingly, the security is 
maintained during all periods of 
beginning [begining]/supplying/end of 
the service. For example, in case the 
registered user is confirmed and 
supplied money through an Automatic 
Teller Machine, or the user operates a 
handy terminal by watching secret 
information on a display, the security 
of the user is effectively protected 
according to the present invention. 
[0043] Next, additional 
embodiments of the present invention 
will be explained. 

(a) ease of monopoly of 
terminal/improvement of common use 
of terminal (second embodiment) 
[0044] In the first embodiment, 
another person can not use the terminal 
while the user is supplied the service 
through the terminal. Therefore, the 
other person must wait to use the same 
terminal until [untill] the first user 
finishes. In short, in the first 



embodiment, use by a specified user 
for the terminal excludes use by 
another [an other] person. 
[0045] In a second embodiment, in 
case the situation for the user not 
capable of using the service is detected 
before the supply of the service for the 
user finishes, the supply of the service 
is reserved (interrupted) and the new 
user can be supplied the service from 
the beginning [begining]. As a result, 
during the interruption of the service, a 
different service is supplied to another 
user. When the original user comes 
back to the terminal, the reserved 
service is supplied to the original user 
again. Therefore, while each service is 
not completed respectively, plural 
users can be supplied each service 
from the same terminal one after 
another. Additionally, if the original 
user does not come back to the same 
terminal, the reserved service is 
finished and memory area 
corresponding to the reserved service 
is relieved to save memory resources. 
[0046] FIG. 4 is a flow chart of 
processing of the service supply 
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permission section according to the 
second embodiment. In this case, 
assume that a computer network is 
commonly used by plural users and a 
plurality of terminals connected to the 
network provide a security function. 
In FIG. 4. steps S15, S16 5 S17, S18 are 
newly added in comparison with FIG. 
3. Each user is confirmed respectively 
and supplied the service through a 
respective terminal (step SI, S2). 
[0047] In this place, assume that a 
user A creates a document using a 
terminal A and leaves from the 
terminal A without completing the 
work. The terminal A detects that the 
user A is not using the service ("No" at 
step S8) and that the supply of the 
service is not completed ("No" at step 
SI 5). The service of creating the 
document at the present time is stored 
(reserved) in a memory area of the 
terminal and the supply of the terminal 
is interrupted (step SI 6, SI 7). In this 
case, the situation for the user not 
using the service is detected according 
to the use situation information D3. 
Completion situation of the supply of 



the service is decided by completion 
notice sent by the user. Next, a user B 
is confirmed by the terminal [teminal] 
A and works through the terminal A. 
Assume that when the user A comes 
back to the terminal A, the user B goes 
away from the terminal A. The 
terminal A newly reserves the results 
of the service for the user B (steps S8 5 
SI 5, SI 6, SI 7) and begins the supply 
of the reserved service to the user 
A(steps S8, Sll, S13. 514). The 
memory area to store the result of the 
service is commonly used by each 
terminal on the network. Therefore, if 
the user B is confirmed by a terminal 
B (steps S4, S6. S7), the reserved 
result of the service for the user B is 
supplied through the terminal B (steps 
S8 5 S11,S13, S14). On the other hand, 
if the user A does not come back to the 
terminal A during a predetermined 
time ("Yes" at step SI 8), the terminal 
A finishes the service of the user A 
immediately and relieves the memory 
area corresponding to the service result 
of the user A (steps S9, SI 0). In this 
way. a situation of storing many 
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service results in memory area is 
prevented. The predetermined time as 
decision [decision] standard at step 
SI 8 is freely set by the kind of the 
service. 

[0048] As mentioned above, in the 
second embodiment, the supply of the 
service for each user is reserved until 
the work of each user is completed. In 
short, a right service-use generated by 
confirming the user is assured until 
[untill] the user's work is completed. 
The user B does not have to wait until 
the user A finishes using the terminal. 
By changing the user A for the user B, 
the user B can use the terminal without 
infringement for the security of the 
user A. 

(b) Extension [Extention] for 
unspecified user (third embodiment) 
[0049] In the first embodiment, in 
order to be supplied the service, the 
user must be confirmed as a registered 
person. In short, if the user is not 
previously registered, he can not be 
supplied the service. However, there 
is a service to be used by many and 
unspecified persons. Therefore, in a 



third embodiment, the service is 
supplied to an unspecified person 
without confirmation. At that time, a 
comparison dictionary of the 
unspecified person is generated and 
registered. During the supply of the 
service, the user (the unspecified 
person) is discriminated from other 
users by referring to the comparison 
dictionary. Therefore, the service is 
supplied to many and unspecified 
persons while the security is 
maintained. 

[0050] FIG. 5 is a flow chart of 
processing of the service supply 
permission section 2 according to the 
third embodiment. In FIG. 5. step S19 
is newly added instead of steps S4, S5 
in comparison with FIG. 3. When the 
supply of the service is begun ("No" at 
step S2), the service supply permission 
section 2 detects the person in the 
service use area according to the 
person detection/non-detection 
information Dl (step S3). If the 
person is detected ("Yes" at step S3), 
the service supply permission section 2 
generates the person comparison 
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dictionary (step SI 9) and registers the 
person as a new user (step S6). After 
that, the use situation of the new user 
is continuously monitored, and the 
service supply permission section 2 
begins to supply the service (step S7). 
In this way, the service is supplied to 
an unspecified person without 
confirmation and the person 
comparison dictionary is generated to 
be registered. Therefore, a new user 
except for the pre-registered user is 
able to be supplied the service. In this 
case, during the supply of the service 
for the new user, he is discriminated 
from another person by referring the 
person comparison dictionary in order 
to maintain the security. 

(c) Warning of infringement 
(fourth embodiment) 
[0051] In the fourth embodiment, 
when an infringement situation of the 
security is detected, a warning is sent 
to the user. In this case, the service 
supply permission section 2 control 
[controlls] the beginning [beginning] 
and end of the service and does not 
execute interruption and cancellation 



of the service in case of infringement 
of the security. 

[0052] FIG. 6 is a flow chart of 
processing of the service supply 
permission section 2 according to the 
fourth embodiment. In FIG. 6, step 
S20 is a newly added step instead of 
step S12 in comparison with FIG. 3. 
First, when the situation detection 
section 1 detects the infringement 
situation of non-user ("Yes" at step 
Sll), the service supply permission 
section 2 sends a warning of the 
infringement to the user (step S20). 
Concretely speaking, the warning is 
outputted through an image (warning 
sentence/mark, controlling of color! 
brightness/blink) or sound (speaking of 
warning sentence/warning buzzer). 
When the user receives the warning, it 
is possible for the user to protect his 
security by, for example, turning off a 
power supply of the terminal or hiding 
a display of the terminal. 

(d) Interactive controlling in 
case of warning (fifth embodiment). 
[0053] In the fifth embodiment, by 
using input means (key board) and 
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output means (display) the supply of 
the service is controlled by the user's 
operation for the warning. FIG. 7. is a 
block diagram of the security 
apparatus according to the fifth 
embodiment. In FIG. 7, an interactive 
control section 4 is a new section in 
comparison with FIG. 1 . FIG. 8 is a 
flow chart of processing of the service 
supply permission section 2 according 
to the fifth embodiment. In FIG. 8. 
steps S20, S21 are new added steps in 
comparison with FIG. 3. 
[0054] The interactive control 
section 4 receives an indication of the 
user for the warning of the 
infringement and contrails the supply 
of the service according to the 
indication. The kind of the indication 
is continuation/interruption/end of the 
service. The user selects the kind of 
the indication by. for example using a 
keyboard. In FIG. 8, during the supply 
of the service, if the situation detection 
section 1 detects infringement by a 
non-user ("Yes" at step Sll) 5 the 
service supply permission section 2 
sends the warning to the user (step 



S20). After sending the warning, the 
interactive control section 4 receives 
an indication of the user (step S21), the 
service supply permission section 2 
controls [contrails] the supply of the 
service according to the indication. 
[0055] If the user indicates 
continuation of the service 
("CONTINUATION" at step S21), the 
interactive control section 4 outputs 
the indication of continuation to the 
service supply permission section 2. 
The service supply section 3 continues 
to supply the service according to the 
indication of continuation. 
[0056] If the user indicates 
interruption of the service 
("INTERRUPTION" at step S21), the 
interactive control section 4 outputs 
the indication of interruption to the 
service supply permission section 2. 
The service supply section 3 interrupts 
the supply of the service according to 
the indication of the interruption. 
[0057] If the user indicates end of 
the service ("END" at step S21), the 
interactive [interractive] control 
section 4 outputs the indication of end 
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to the service supply permission 
section 2. The service supply section 3 
finishes the supply of the service. 
[0058] In this way, when the user 
receives the warning of infringement, 
the user can select 

continuation/interruption/end of 
service by using interactive function. 
As a result, if the user works through 
equipment in an office, whenever an 
office colleague (person not necessary 
for security) stands behind the user, 
interruption of the work is excluded. 
In short an interruption , [a change for 
the worse] of use [-feeling] of the 
equipment is avoided. Otherwise, 
when using the equipment in a public 
place, interruption of the service is 
indicated by the user for the warning 
of a non-user standing behind the user. 
[0059] In the fifth embodiment, the 
user can select the indication to keep 
security in case of infringement of the 
security. After selecting the 
indication, the apparatus (equipment) 
can automatically keep the security of 
the user. In this case, if the warning of 
the infringement is outputted as an 



image, the user understands 
circumstances around the user by 
watching the image (the user and non- 
user are included). The user can select 
the indication 
(continuation/interruption/end of the 
service) according to the 
circumstances. 

(e) Default action of neglecting 
the warning (sixth embodiment). 

In the sixth embodiment, in 
case the indication is not selected by 
the user within predetermined time 
after generating the warning, the 
apparatus automatically executes 
predetermined (default) controlling of 
the supply of the service. This 
predetermined controlling is freely set 
by the user. 

[0060] FIG. 9 is a flow chart of 
processing of the service supply 
permission section 2. In FIG. 9, steps 
S20, S21, S22, S23, and S24 are added 
steps in comparison with FIG. 3. The 
construction of the security apparatus 
of the fifth embodiment is the same as 
in FIG. 7. In FIG. 9. during the supply 
of the service, if the situation detection 
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section 1 detects an infringement 
situation ("Yes" at step Sll), the 
service supply permission section 2 
sends the warning to the user (step 
S20). After the warning, the 
interactive control section 4 receives 
indication of the user. In this case, if 
no indication of the user is received 
above predetermined time (step S22 
and S23), the interactive control 
section 4 outputs the purport of no 
indication to the service supply 
permission section 2. The service 
supply permission section 2 controls 
[controlls] the supply of the service 
according to a predetermined control 
(step S24). For example, if 
continuation of the service is set as the 
predetermined control 
("CONTINUATION" at step S24), the 
service supply section 3 continues to 
supply the service. If interruption of 
the service is set as the predetermined 
control ("INTERRUPTION" at step 
S24), the service supply section 3 
interrupts the supply of the service. If 
end of the service is set as the 
predetermined controlling ("END" at 



step S24) the service supply section 3 
finishes the supply of the service. On 
the other hand, if the indication of the 
user is received within the 
predetermined time ("Yes" at step 
S22), the interactive control section 4 
receives the indication (step S21) and 
the service supply permission section 2 
control [controlls] the supply of the 
service according to the indication. In 
this case, if the user indicates 
continuation of the service 
("CONTINUATION" at step S21), the 
interactive control section 4 outputs 
the continuation indication to the 
service supply permission section 2. 
The service supply section 3 continues 
to supply the service. If the user 
indicates interruption of the service 
("INTERRUPTION" at step S21), the 
interactive control section 4 outputs 
the interruption indication to the 
service supply permission section 2. 
The service supply section 3 interrupts 
the supply of the service. If the user 
indicates end of the service ("END" at 
step S2 1 ) 5 the interactive control 
section 4 outputs the end indication to 



Library: Los Angeles; Document #: 40165vl 
43701.00023 



-16- 



the service supply permission section 
2. The service supply section 3 
finishes the supply of the service, 
[0061] In this way if the user 
neglects a warning of infringement. 
The security of the user is kept 
according to a predetermined control 
(action of default). Otherwise, it is 
possible to control the supply of the 
service different form the 
predetermined control for some 
reasons by a user's indication in case 
of the warning. For example, in an 
office, as for the warning generated 
whenever a colleague stands behind 
the user operating the equipment. The 
user does not have to indicate 
continuation of the service. The 
predetermined control is already set as 
continuation of the service. In this 
case, the user can continue the 
operation while neglecting the 
warning. In case the user operates the 
equipment in a public place, the 
predetermined control is set as 
interruption of the service. In this 
case, security of the user for the 



equipment is protected while he 
neglects the warning. 

(f) Security degree by unit of 
service/information (seventh 
embodiment) 

[0062] In the above embodiments, 
predetermined security is activated for 
service or information. However, 
security degree is actually different by 
unit of service/information. If a fixed 
security degree is activated, useless 
interruption of the service is often 
executed. This happens in case of 
plural workers using information 
equipment. Accordingly, it is desired 
that the security is activated according 
to the kind of the service/information. 
For example, assume that electric mail 
is transmitted using a PC (personal 
computer) or a document is created 
using WP (word processor). Plural 
services are then supplied on the 
display of the equipment. The electric 
mail represents a private letter and its 
security degree is high. On the other 
hand, the security degree of the 
document by WP differs according to 
the kind of the document. While the 
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user creates the document whose 
security degree is low, the security 
degree of the document grows by 
adding another important document. 
In this way, the security degree is 
dynamically changed according to the 
kind of the service/information. 
[0063] In the seventh embodiment, 
the security degree is changed by unit 
of the service/information. FIG. 10 is a 
flow chart of processing of the service 
supply permission section 2 according 
to the seventh embodiment. In FIG. 
10, step S25 added between SI 1 and 
S 12 is a different step in comparison 
with FIG. 3. During the supply of the 
service, if the situation detection 
section 2 detects infringement of the 
security ("Yes" at step Sll), the 
service supply permission section 2 
controls [controlls] the supply of the 
service according to the security 
degree set by the kind of 
service/information (step S25). In this 
case, it may be possible that the 
security degree is set to a mail tool to 
display the mail but not set to a WP 
tool. For example, when another 



person watches the display behind the 
user, the mail tool is only interrupted 
(the mall window is closed) and 
the WP window is continuously 
opened. 

[0064] As for setting the security 
degree, for example, the user sets 
disclosure-possible at the head portion 
of mail to be sent. A window showing 
the mail tool does not react to the 
infringement of the security. As for 
both sending side or receiving side, the 
mail is able to remain disclosing. 
[0065] If the user indicates a strict 
secret at the header of a document, the 
WP window to display the document 
reacts to the infringement of security. 
For example, the security [sequrity] 
degree for creating the document is 
dynamically changed by adding other 
important document. In this way, even 
if the service is supplied to the same 
object, the supply of the service is 
interrupted or continued according to 
the content (for example, regular 
document or strict secret document). 

(g) detection of visual line 1 
(eighth embodiment) 
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[0066] In the above embodiments, 
when another person watches the 
display behind the user creating a 
document, the infringement of security 
of the user is detected. However, it is 
not necessary to pay attention to the 
other person, but to pay attention to the 
visual line (or line of sight) of the user. 
For example, when the visual line of 
the user leaves the screen of the 
display, the window may be closed. In 
the eighth embodiment, by detecting a 
visual line of the user as use situation, 
the supply of the service is interrupted 
when the visual line leaves the screen. 
[0067] FIG. 11 is a block diagram 
of the situation detection section 1 
according to the eighth embodiment. 
A visual line detection section 16 is an 
added section in comparison with FIG. 
2. The visual line detection section 16 
detects movement of the visual line of 
the user in case the user (registered) is 
included in the person discrimination 
information D2. For example, the face 
area of the user is extracted from the 
image and the eye area is extracted 
from the face area. The visual line is 



pursued according to the eye area. In 
this case, if a direction of the visual 
line is toward a predetermined area 
(visual area of the screen), an 
indication is outputted. If the direction 
of the visual line is not toward the 
predetermined area, another indication 
is outputted. These indications 
represent visual line information D5. 
[0068] In this way, in the eighth 
embodiment, the visual line 
information D5 is outputted to the 
service supply permission section 2 as 
one of the person information. FIG. 12 
is a flow chat of processing of the 
service supply permission section 2 
according to the eighth embodiment. 
In FIG. 12, step S26 is added instead 
of step SI 1 in comparison with FIG. 3. 
During the supply of the service, 
assume that the situation detection 
section 1 detects that the user is under 
a situation to be supplied the service 
("Yes" at step S8). In this case, the 
visual line detection section 16 keeps 
watch for the movement of the visual 
line of the user. If the direction of the 
visual line departs from the 
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predetermined area of the display 
("Yes" at step S26), the situation 
detection section 1 outputs the visual 
line information D5 to the service 
supply permission section 2. The 
service supply permission section 2 
controls the service supply section 3 to 
interrupt the supply of the service (step 
S12). FIG. 13 is a schematic diagram 
of detection of the visual lines of the 
user in the image according to the 
eighth embodiment. As shown in FIG. 
13, the visual lines of the user are 
detected from position of black eyes of 
the user's face in the image. 
[0069] In this way, in the eighth 
embodiment, detection of the visual 
line of the user controlls the supply of 
the service. Therefore, when another 
person walks behind the user, security 
of the service is kept only if the visual 
line of the user departs from the 
display. 

(h) Detection of visual line 2 
(ninth embodiment) 
[0070] In the eighth embodiment, a 
direction of the visual line of the user 
controls [controlls] the supply of the 



service. However, if another person 
looks furtively at the display behind 
the user, the supply of the service can 
not be controlled. If the user turns his 
eyes away from the display whenever 
the other person walks behind the user, 
the supply of the service is interrupted 
to every time. Therefore, in the ninth 
embodiment, detection of the visual 
line of the non-user controls [controlls] 
the supply of the service. 
[0071] FIG. 14 is a flow chart of 
processing of the service supply 
permission section 2 according to the 
ninth embodiment. In FIG. 14, step 
S27 is added step between step Sll 
and step S12 in comparison with FIG. 
3. In this case, the visual line 
detection section 16 detects the visual 
line of the non-user. In case the person 
is not discriminated by referring to the 
person discrimination information D2 
(the person is decided to be a non- 
user), the movement of the visual line 
of the non-user is detected. In the 
same way as in the eighth 
embodiment, the face area is extracted 
from the image including the non-user 



Library: Los Angeles; Document #: 40165vl 
4370L00023 



-20- 



and the eye area is extracted from the 
face area. The visual line of the eye 
area is pursued. If a direction of the 
visual line of the non-user points to a 
predetermined area (visible area of the 
display), an indication is outputted. If 
the direction of the visual line of the 
non-user does not point to the 
predetermined area, another indication 
is outputted. This purport is 
represented as the visual line 
information D5. During the supply of 
the service, assume that the situation 
detection section 1 detects that the 
non-user infringes the security of the 
non-user ("Yes" at step Sll). In this 
case, the visual line detection section 
16 keeps watch for the move of the 
visual line of the user. If the direction 
of the visual line of the non-user points 
to the predetermined area, for example, 
the non-user watches the display 
behind the user ("No" at step S27), the 
situation detection section 1 outputs 
the visual line information to the 
service supply permission section 2. 
The service supply permission section 
2 controls [contrails] the service 



supply section 3 to interrupt the supply 
of the service. FIG. 15 is a schematic 
diagram of detection of the visual lines 
of the non-user in the image according 
to the ninth embodiment. As shown in 
FIG. 15, the visual lines of the non- 
user are detected from position of 
black eyes of the non-user's face in the 
image. 

[0072] In this way, in the ninth 
embodiment, the detection of the 
visual line of the non-user controls 
[contrails] the supply of the service. 
Therefore, in case the non-user looks 
furtively at the display behind the user, 
the security of the service for the user 
is kept. 

(i) Detection of visual line 3 
(tenth embodiment) 
[0073] In the tenth embodiment, 
detection of the visual line of the user 
and the non-user controls [contrails] 
the supply of the service. FIG. 16 is a 
flow chart of processing of the service 
supply permission section 2 according 
to the tenth embodiment. In FIG. 16, 
step S26 and step S27 are added steps 
in comparison with FIG. 3. During the 
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supply of the service, assume that the 
situation detection section 1 detects 
that the user is under a situation to be 
supplied the service ("Yes" at step S8). 
In this case, the visual line detection 
section 16 detects movement of the 
visual line of the user. If the visual 
line of the user departs from the 
predetermined area, for example, the 
user turns his eyes away from the 
display ("Yes" at step S26), the 
situation detection section 1 outputs 
the visual line information D5 to the 
service supply permission section 2. 
Then, the service supply permission 
section 2 controls [controlls] the 
service supply section 3 to interrupt 
the supply of the service (step SI 2). 
[0074] On the other hand, assume 
that the direction of the user points to 
the predetermined area of the display 
(the user does not turn his eyes away 
from the display) ("No" at step S26) 
and the situation detection section 1 
detects that the non-user infringes the 
service of the user ("Yes" at step SI 1). 
In this case, the visual line detection 
section 16 keeps watch for movement 



of the visual line of the non-user. If 
the direction of the visual line of the 
non-user points to the predetermined 
area of the display (the non-user 
watches the display behind the user) 
("No" at step S27), the situation 
detection section 1 outputs the visual 
line information to the service supply 
permission section 2. The service 
supply permission section 2 controls 
[controlls] the service supply section 3 
to interrupt the supply of the service 
(step SI 2). 

[0075] In this way, in the tenth 
embodiment, detection of the visual 
line of the user and the non-user 
controls [controlls] the supply of the 
service. Therefore, the security of the 
service for the user is strengthened. 
[0076] In the eighth, ninth, tenth 
embodiments, the visual line of the 
user or the non-user is detected. 
However, by detecting a direction of a 
face of the user or non-user, the supply 
of the service may be controlled. In 
this case, as for the user, if the 
direction of the user's face does not 
point to the display, the supply of the 
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service is interrupted. As for the non- 
user, if the direction of the non-user's 
face points to the display, the supply of 
the service is interrupted. As for the 
method for detecting the direction of 
the face, the face area is extracted from 
the image including the user or the 
non-user. The direction of the face is 
pursued according to the face area. 
[0077] In the above embodiments, 
the image is used for detecting the 
person information. However, sound 
or wireless may be used for detecting 
the person information. In case of the 
sound, the user is specified by the 
user's voice existing in the service-use 
area. If the user' s voice is 
continuously detected, it is decided 
that the user is under a situation to be 
supplied the service. If a voice other 
than the user's is detected, it is decided 
that the security of the service of the 
user is infringed. In case of the 
wireless, the user is specified by 
detecting ID information of a wireless 
card. The user previously brings the 
wireless card with him. If the ID 
information of the user's wireless card 



is continuously detected, it is decided 
that the user is under a situation to be 
supplied the service. If ID information 
other than the user's wireless card is 
detected, it is decided that the security 
[sequrity] of the service of the user is 
infringed. 

[0078] FIG. 17 is a schematic 
diagram showing a method of 
supplying a software for the present 
invention. A series of processings of 
the service supply permission section 2 
(shown in FIGS. 3-6, 8-10, 12, 14, 16) 
and necessary information may be 
stored in a memory device as shown in 
FIG. 17. In this case, it is possible that 
the memory device is applied for each 
apparatus or content of the memory 
device is transmitted to each apparatus 
by a communication 5 device. 
[0079] A memory can be used to 
store instructions for performing the 
process described above, such a 
memory can be a CD-ROM, floppy 
disk, hard disk, magnetic tape, 
semiconductor memory, and so on. 
[0080] Other embodiments of the 
invention will be apparent to those 
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skilled in the art from consideration of 
the specification and practice of the 
invention disclosed herein. It is 
intended that the specification and 
examples be considered as exemplary 
only, with the true scope and spirit of 
the invention being indicated by the 
following claims. 
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